Modeling the Java Bytecode Verifier

The Java programming language has been widely described as secure by design. Nevertheless, a number of serious security vulnerabilities have been discovered in Java, particularly in the Bytecode Verifier, a critical component used to verify class semantics before loading is complete. This paper describes a method for representing Java security constraints using the Alloy modeling language. It further describes a system for performing a security analysis on any block of Java bytecodes by converting these bytecodes into relation initializers in Alloy. Any counterexamples found by the Alloy analyzer correspond directly to potentially insecure code. Analysis of the approach is provided in the context of known security exploits, including type confusion attacks, invalid memory accesses and control flow misdirection. This type of analysis represents a significant departure from standard malware analysis methods based on signatures or anomaly detection.